Risk-of-bias assessment using Cochrane's revised tool for randomized trials (RoB 2) was useful but challenging and resource-intensive: observations from a systematic review.

OBJECTIVES: To report our experience using version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2). STUDY DESIGN AND SETTING: Two reviewers independently applied RoB 2 to results of interest in a large systematic review of complex interventions and reached consensus. We recorded the time taken, and noted and discussed our difficulties using the tool, and the resolutions we adopted. We explored the time taken with regression analysis and summarized our experience of implementing the tool. RESULTS: We assessed risk of bias in 860 results of interest in 113 studies. Staff resource averaged 358 minutes per study (SD 183). Number of results (ß = 22) and reports (ß = 14) per study and experience of the team (ß = -6) significantly affected assessment time. To implement the tool consistently, we developed cut points for missingness and considerations of balance regarding missingness, assumed some concerns with intervention deviations unless otherwise prevented or investigated, some concerns with measurements from unblinded self-reporting participants, and judged low risk of selection for certain dichotomous outcomes despite the absence of an analysis plan. CONCLUSION: The RoB 2 tool and guidance are useful but resource-intensive and challenging to implement. Critical appraisal tools and reporting guidelines should detail risk of bias implementation. Improved guidance focusing on implementation could assist reviewers.